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FOREWORD 


This  research  and  development  was  conducted  in  response  to 
Navy  Decision  Coordinating  Paper,  Performance  Aids  Test  and 
Evaluation  (NDCP-Z0828-PN) , under  the  sponsorship  of  the  Deputy 
Chief  of  Naval  Operations  (OP-01) . The  objectives  of  the  NDCP 
are  to  define  the  state  of  the  art  in  Job  Performance  Aids  (JPA) 
technology,  to  develop  a conceptual  model  for  an  integrated  JPA- 
based  personnel  system  including  cost  benefits  and  trade-off 
analysis,  to  test  the  JPA  concept,  and  to  quantify  performance 
increments  and  cost  benefits  obtainable  for  various  applications. 

The  Performance  Aids  Test  and  Evaluation  project  is  based  on 
the  premise  that  current  Navy  personnel  problems  have  their 
roots  in  the  personnel  system  itself  and  therefore  can  best  be 
addressed  through  a total  "systems  approach."  Therefore,  an 
Integrated  Personnel  Systems  Approach  (IPSA)  was  devised  which 
is  based  on  joint  consideration  of  training  design,  JPA  design, 
job  design,  career  structures,  advancement  channels,  incentives, 
and  the  quality  and  quantity  of  personnel  resources  available. 
Coupled  with  a cost  trade-off  model,  IPSA  is  being  employed  to 
evolve  a new  personnel  system  concept  entitled  Enlisted  Personnel 
Individualized  Career  System,  or  "EPICS." 

The  effort  described  in  this  report  evaluated  the  potential 
that  some  unique  forms  of  performance  aids  have  for  an  integrated 
JPA-based  personnel  system.  The  performance  aids  evaluated  in 
this  project  are  referred  to  as  hybrid  aids  and  enriched  hybrid 
aids.  In  evaluating  both  types,  concern  focused  on:  (1)  whether 
Navy  technicians  would  rely  more  heavily  on  the  new  aid  forms  as 
compared  to  conventional  aids,  and  (2)  whether  technicians  would 
experience  any  learning  enhancement  as  a result  of  the  new  aid 
forms.  The  evaluation  comprised  a pilot  study  and  the  collection 
of  expert  and  user  opinion  data.  The  experimental  results  and 
opinion  data  were  used  as  the  basis  for  selecting  the  most 
promising  enriched  hybrid  aids  for  more  rigorous  testing.- 

“4  . 

* \ 

This  is  the  fifth  in  a series  of  reports  dealing  with  the  ... 
NDCP.  The  first,  tiPRDC  TR  77-3,  included  seven  papers  assessing 
the  state  of  the  art  in  JPA  technology.  The  second,  NPRDC  TN 
78-6,  described  a preliminary  attempt  to  define  an  enlisted 
personnel  system  concept,  with  major  emphasis  on  the  use  of  job 
performance  aiding.  The  third,  NPRDC  TR  78-26,  dealt  with  the 
systematic  review  and  organization  of  existing  JPA  techniques, 
related  research  data,  and  various  applicable  principles  and 
concepts.  The  fourth,  NPRDC  TN  79-1,  is  a preliminary  attempt 
to  define  a JPA  selection  algorithm  for  an  Integrated  Personnel 


System  (IPS) . Subsequent  reports  will  deal  with  a series  of 
tasks  involving  the  development  of  an  experimental  IPS  that  can 
be  implemented  and  evaluated  in  an  operational  setting . 

The  technical  monitor  was  Dr.  Robert  J.  Smillie. 


Donald  F.  Parker 
Commanding  Officer 


SUMMARY 


Problem 

Research  has  shown  job  performance  aids  (JPAs)  to  be  a 
cost-effective  means  of  facilitating  performance  of  maintenance 
technicians,  especially  technicians  with  little  on-the-job 
experience.  As  a major  component  of  a Navy  personnel  system, 
however,  JPAs  must  provide  the  JPA  user  with  career  support. 

The  current  use  of  JPA  technology  does  not  provide  the  user  with 
total-career  support. 

Objective 

The  objective  of  this  effort  was  to  conceptualize  and 
evaluate  JPA  candidates  (troubleshooting)  for  an  expanded  role 
that: 


1.  Results  in  early  productivity  by  inexperienced  tech- 
nicians, despite  minimum  "front  end"  training. 

2.  Allows  JPAs  to  be  usable  at  all  stages  of  a technician's 
career,  not  merely  in  the  initial  stages. 

3.  Results  in  or  encourages  the  learning  of  knowledge 
required  to  qualify  for  career  milestones. 

Approach 

The  concepts  of  "hybrid  aids"  and  "enriched  aids"  were 
investigated  to  determine  their  applicability  to  the  objectives 
for  JPAs.  Through  pilot  testing,  the  hybrid  aids  were  evaluated 
in  terms  of  their  capacity  for  allowing  a technician  to  graduate 
over  a period  of  use,  from  a reliance  on  the  "directive  JPA"  to 
the  more  independent,  decision-making  mode  of  the  "deductive 
JPA."  Expert  judgment  determined  which  of  the  enriched  JPAs 
were  effective  in  imparting  significant  kinds  of  incidental 
learning  during  transition  from  the  instruction- following  to  the 
autonomous  mode. 

Findings 

1.  The  use  of  hybrid  JPAs  for  troubleshooting  tasks  appears 
feasible. 

2.  Enrichment  appears  feasible  when  used  to  enhance  the 
transition  effect  of  hybridization. 
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3.  Enrichment  appears  feasible  when  used  to  convey  career- 
relevant knowledge  about  the  hardware  and  the  operational 
principles  that  underlie  classes  of  hardware. 

Recommendations 

1.  Findings  in  this  study  were  derived  from  (a)  pilot 
testing  (e.g.,  small  sample  size,  nonrepresentative  subjects) 
and  (b)  opinions  of  experts  in  both  JPA  research  and  JPA  use. 
Before  conducting  any  extensive  field  test  of  the  enriched 
hybrid  JPA,  a more  rigorous  demonstration  of  its  feasibility 
is  required. 

2.  Analytical  work  is  required  toward  application  of  the 
enriched,  hybrid  JPAs  investigated  in  this  project.  Key 
features  needing  further  definition  include  format  considerations, 
exposure  control,  and  interfaces  with  other  features  of  the 
personnel  system  (e.g.,  training,  performance  assessment). 
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INTRODUCTION 


Problem 


Job  performance  aid  (JPA)  research  in  the  past  has 
concentrated  on  two  basic  JPA  formats — directive  (e.g.,  fully 
proceduralized  JPAs)  and  deductive  (e.g.,  schematics).  Directive 
formats  allow  apprentice  technicians  (novices)  to  perform  on  the 
job  without  extensive  training  and  experience,  while  deductive 
formats  allow  system  technicians  (fully  trained  and  experienced 
individuals)  to  continue  in  the  job  without  relying  on  memory 
alone.  Neither  format  is  totally  satisfactory  for  the  individual 
at  an  intermediate  level  of  skill  (e.g.,  subsystem  technician). 
The  deductive  JPAs  are  still  too  complicated,  and  the  directive 
JPAs  are  inadequate  and  too  time  consuming  for  complex  trouble- 
shooting. Therefore,  an  innovative  JPA  format  is  needed  to 
bridge  the  gap  between  apprentice  and  experienced  technicians. 

Purpose 

The  purpose  of  the  present  project  was  to  conceptualize  those 
JPA  candidates  (troubleshooting)  for  subsystem  technicians  that 
bridge  the  gap.  In  addition,  the  candidate  JPA  concepts  were 
evaluated  to  select  those  candidates  that  warrant  additional 
verification  testing. 

The  development  of  this  innovative  JPA  will  occur  in  three 
stages : 

1.  Conceptualization  and  evaluation  (present  project). 

2.  Verification  testing  (JPAs  independent  of  other  elements 
of  the  personnel  system) . 

3.  Shipboard  demonstration  (JPAs  integrated  with  the  other 
elements  of  the  personnel  system) . 

Background  v 

The  present  project  is  part  of  an  overall  program  the 
objective  of  which  is  to  design  and  demonstrate  the  benefits  of 
a JPA-based  Integrated  Personnel  System  (IPS)  (Blanchard  & 

Laabs,  1978).  The.  characteristics  of  the  JPAs  in  the  IPS  are: 

1.  Use  of  Jt’As  results  in  early  productivity  by  inexperi- 
enced technicians  despite  minimum  front-end  training. 
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THE  EVOLUTION  OF  JOB  PERFORMANCE  AIDS 


Proceduralized  Aids 

Designed  to  support  maintenance,  the  classical  Technical 
Manual  (TM)  contains  descriptions  of  the  hardware  and  of  the 
procedures  necessary  to  operate  and  maintain  the  hardware. 
Traditionally,  technicians  used  the  TM  to  gain  an  understanding 
of  the  system  to  supplement  or  substitute  for  the  relatively 
brief  TM  procedures.  As  systems  grew  more  complex,  however, 
this  approach  became  very  time-consuming  and  resource-intensive. 
Extensive  front-end  training  was  necessary  for  a new  technician 
to  understand  a system.  Even  when  equipped  with  this  training, 
new  technicians  required  lengthy  apprenticeships  on  the  job 
before  they  were  capable  of  independent  maintenance  performance. 

In  the  late  1950s,  the  Air  Force  initiated  research  (Hoehn  & 
Lumsdaine,  1958)  designed  to  emphasize  the  procedural  portions 
of  the  TM.  The  intention  of  this  early  research  was  to  find  an 
approach  that  shortened  both  the  apprenticeship  period  and  the 
front-end  training.  Experiments  using  proceduralized  aids  gave 
promising  results,  and  eventually  led  to  field  demonstrations  of 
what  have  become  known  as  job  performance  aids  (JPAs) . The  best 
known  of  these  quite  successful  demonstrations  were  PIMO — 
Presentation  of  Information  for  Maintenance  and  Operations  (Goff, 
Schlesinger,  & Parlog,  1969)  and  AMSAS — Advanced  Manpower 
Concepts  for  Sea-Based  Aviation  Systems  (Post  & Brooks,  1970). 

In  spite  of  these  apparent  successes,  the  military  did  not 
adopt  the  JPA  form  of  TM.  JPA  research  continued  on  a small 
scale  until  1974,  when  Malehorne  (1975)  formulated  a major 
advance  in  JPA  technology.  Malehorne  proposed  that  JPAs  would 
not  be  accepted  until  they  were  shown  to  be  capable  of  working 
on  a continuing  basis  with  the  Navy's  current  personnel  system. 
The  following  is  one  of  several  examples  he  presented  to 
illustrate  the  impracticality  of  existing  JPA  technology: 

Advocates  claim  that  the  use  of  JPAs  will  allow 
extensive  reductions  in  theory-based,  front  end 
training  without  adversely  affecting  the  early 
productive  labor  obtainable  from  a new  technician. 
However,  the  advocates  failed  to  recognize  that 
reducing  the  theory-based  training  also  seriously 
erodes  the  technician's  ability  to  pass  advance- 
ment in  Rate  exams  which,  at  the  present  time, 
demand  extensive  knowledge  of  theory.  (p.  142) 
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In  addition  to  these  types  of  JPA  concerns,  questions  arose 
about  the  technician's  acceptance  of  the  proceduralized  form  of 
documentation.  It  was  felt  that  technicians  would  quickly  resent 
(and  therefore  fail  to  use)  the  proceduralized  JPA  if  it  were 
the  only  form  of  documentation  available.  In  other  words,  some 
changes  would  have  to  be  made  in  the  JPA  to  integrate  it  into  the 
larger  Personnel  System  and  to  enhance  its  acceptability.  One  } 

JPA  approach,  called  "hybridizing,"  seemed  to  have  the  potential 
to  address  at  least  some  of  these  needs. 

History  of  the  Hybrid  Aid 

Concept 

The  hybrid  aid  was  conceived  in  projects  sponsored  by 
the  Naval  Air  Systems  Command  (Post  & Price,  1972;  Post  & Price, 

1973) . The  concept  emerged  from  a review  of  the  types  of 
troubleshooting  aids  included  in  26  information  presentation 
techniques. ^ This  review  concluded  that  the  terms  "directive"  or 
"deductive"  could  characterize  all  the  troubleshooting  approaches 
used  by  the  presentation  techniques.  Table  1,  which  was  adapted 
from  Post  and  Price  (1973),  summarizes  these  characterizations. 

Concerned  with  improving  what  they  perceived  to  be  poor 
aid  acceptance,  Post  and  Price  examined  the  use  of  directive  and 
deductive  aid  forms  against  three  criteria  drawn  from  the  litera- 
ture on  work  and  job  satisfaction.  As  researched  by  Ford  (1969) , 
these  criteria  appeared  to  be  particularly  relevant  to  JPA 
acceptance : 

1.  Opportunity  to  Learn.  A good  job  (job  aid)  is  one 
in  which  the  worker  is  afforded  opportunities  to  learn  skills  or 
knowledge  relevant  to  career  advancement. 

2.  Chal lenge . Workers  tend  to  be  satisfied  with  jobs 
(job  aids)  that,  over  a period  of  time,  offer  challenge.  The 
challenge  can  occur  in  the  form  of  quantity  or  complexity  of 
the  tasks  included  in  the  job. 

3.  Meaningf ulness  of  the  Work.  Workers  tend  to  feel 
satisfied  when  they  have  a significant  role  in  performing  tasks 
that  they  perceive  to  be  critical  in  the  output  or  goal  of  their 
organization. 


The  term  deductive  refers  to  those  aid  forms  that  describe 
the  system  (e.g.,  functional  diagrams  or  electronic  schematics). 
The  term  deductive  is  indicative  of  the  behavior  required  of  the 
troubleshooter;  that  is,  he  had  to  understand  the  relationships 
between  systems,  functions,  and  signal  flows  in  order  to  deduce 
the  correct  troubleshooting  procedure. 
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Table  2 summarizes  how  the  use  of  deductive  and 
directive  JPAs  fared  against  these  job  satisfaction  criteria. 

As  shown,  an  examination  was  conducted  for  both  experienced 
and  inexperienced  technicians.  The  conclusion  of  this  analysis 
was  that  the  deductive  JPA  is  the  approach  suitable  for 
experienced  technicians,  since  it  allows  them  to  practice  a 
valued  skill  that  sets  them  apart  from  their  less-experienced 
colleagues.  On  the  other  hand,  the  directive  JPA  appears  to 
benefit  the  inexperienced  technicians,  since  it  allows  them  to 
participate  in  work  that  would  be  beyond  their  capability  under 
conventional  circumstances.  Accordingly,  if  a JPA  is  to  be 
acceptable  to  both  populations,  it  should  involve  some  com- 
bination, or  hybrid  of  the  deductive  and  directive  aid  forms. 


Use  of  Hybrid  JPA 


Scenario  for  Use.  Figure  1 illustrates  the  directive 
and  deductive  elements  of  a hybrid  aid.  The  directive  element 
(presented  alongside  the  deductive  element)  would  provide  the 
inexperienced  technician  with  guidance  to  do  meaningful  work 
that  would  be  beyond  his  capability  with  conventional  TMs . 

The  hybrid  users  would  be  briefed  on  the  objective  of  transi- 
tioning gradually  from  reliance  on  the  directive  element  to 
doing  the  same  task  more  autonomously  by  referring  only  to  the 
deductive  element.  This  transition  is  facilitated  by  the 
side-by-side  presentation  of  the  two  elements  as  well  as  by 
their  design.  For  example,  both  use  the  same  signal  names  and 
test-point  nomenclature,  and  both  cover  the  same  portion  of 
the  hardware.  This  transition  offers  the  JPA  user  a continuing 
challenge.  It  also  offers  an  opportunity  to  learn  career- 
relevant knowledge,  since  mastering  the  use  of  the  deductive 
aid  ensures  mastery  of  many  of  the  content  items  required  to 
pass  the  Personnel  Qualification  System  (PQS).^ 

The  strategy  underlying  the  hybrid  is  that,  over  a 
series  of  performances,  the  technician  will  become  more  and 
more  familiar  with  the  deductive  element  until  he  feels  ready 
to  troubleshoot  using  only  the  deductive  element.  Should  he 
encounter  difficulty  in  using  the  deductive  aid,  he  can  refer 
back  to  the  directive  element  for  guidance  on  how  to  proceed  or 
for  assurance  of  the  correct  response  (i.e.,  does  his  deductive- 
aid  solution  agree  with  the  directive-aid  solution?) . 


When  Introduced.  It  is  appropriate  at  this  point  to 
discuss  the  stage  in  the  technician’s  career  at  which  he  is 
expected  to  be  using  this  form  of  JPA.  We  assume  that  a new 
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The  PQS  is  the  means  by  which  technicians  are  qualified  to 
stand  watch,  which  is,  in  turn,  a prerequisite  for  taking  the 
advancement  in  Rate  exams. 
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Summary  of  Assessing  Directive  and  Deductive  Aid  Forms 
Against  Selected  Job  Satisfaction  Criteria 
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Figure  la.  Illustration  of  a hybrid  aid  (directive  element) 


technician's  first  year  of  duty  would  have  him  performing  non- 
troubleshooting maintenance  (primarily  in  the  preventive 
maintenance  domain)  in  accordance  with  the  instructions  of  a fully 
proceduralized  JPA.  This  experience  will  give  him  considerable 
knowledge  of  test  point  location,  component  nomenclature,  set-up 
and  turn-on  procedures,  and  the  like.  The  knowledge  gained  in 
this  stage  will  equip  him  to  handle  the  relatively  "austere" 
instructions  of  the  directive  element  of  the  hybrid  shown  in 
Figure  1. 

The  length  of  time  required  to  master  the  simple 
input-output  process  of  troubleshooting  depends  on  the  unit's 
troubleshooting  workload  and  manning  level.  For  our  purposes, 
we  are  considering  this  time  period  to  be  on  the  order  of 
6 months  to  a year.  At  this  point,  the  user  of  the  hybrid  aid 
is  a Skill  Level  II  technician,  in  the  terms  of  the  Integrated 
Personnel  System  (Blanchard  & Laabs,  1978) . While  he  has 
mastered  the  simple  logic  method  of  troubleshooting  and  has 
learned  something  of  how  the  system  operates,  he  knows  nothing 
of  the  more  advanced  troubleshooting  processes,  nor  of  the 
theory  underlying  the  operation  of  a system  in  this  class,  which 
is  a major  content  requirement  of  conventional  Rate  advancement 
exams.  The  JPA's  contribution  to  qualifying  the  technicians  on 
these  more  advanced,  theory-based  plateaus  relies  on  the  use  of 
"enriched"  hybrid  JPAs. 

Concept  of  the  Enriched  Hybrid  Aid 

Use  of  the  hybrid  troubleshooting  aid  is  expected  to 
help  the  technician  make  the  transition  from  an  instruction- 
following mode  of  performance  to  a more  autonomous  mode.  Then, 
he  is  required  to  make  decisions  based  on  his  understanding  of  a 
simple-logic,  deductive  troubleshooting  aid.  In  accomplishing 
the  transition,  some  of  the  more  insightful  and  motivated  users 
may  experience  some  incidental  learning.  Enrichment  is  intro- 
duced at  this  point  to  accomplish  either  or  both  of  two  purposes: 

1.  To  explicate  and  formalize  the  learning  opportunities 
inherent  in  the  hybrid  aid,  for  the  benefit  of  the  less  insight- 
ful users. 

2.  To  introduce  learning  possibilities  that  are  related 
to  but  beyond  the  scope  of  the  combined  aid  forms  (e.g.,  theory- 
based  learning) . 

Figure  2 presents  a hybrid  aid  (same  aid  elements  as 
appeared  in  Figure  1)  in  which  enrichment  has  been  added  to  the 
directive  element.  The  enrichment  is  the  information  that  appears 
in  the  brackets.  As  shown,  the  enrichment  message  explains  the 
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troubleshooting  progress  represented  by  the  preceeding  check 
or  test  outcome.  The  explanations  use  the  terminology  of  the 
deductive  aid  as  an  attention-directing  device.  In  this  example, 
the  enrichment  message  is  emphasizing  a subtle  possibility  already 
present  in  the  hybrid  aid.  Other  illustrations  will  be  presented 
later  in  which  the  enrichment  information  is  related  to  but  beyond 
the  scope  of  the  hybrid  aid  content. 

The  major  features  of  the  enrichment  concept  are  summa- 
rized below: 


1.  It  provides  the  user  with  information  to  explicate 

a subtle  characteristic  of  the  hybrid  JPA,  or  one  that  is  related 
to  but  beyond  the  content  of  the  hybrid. 

2.  It  uses  the  task/work  situation  to  lend  meaning  to 
what  might  otherwise  be  perceived  by  the  user  as  abstract  material. 

3.  It  motivates  the  user  as  a result  of  learning  oppor- 
tunities that  are  oriented  to  performance  (e.g.,  to  learn  a 
simple  logic  troubleshooting  method)  or  to  knowledge  (e.g., 
learn  more  about  this  hardware  or  this  class  of  hardware) . 


to 


DEVELOPMENT  OF  THE  HYBRID 
AND  ENRICHED  HYBRID  FORMS  OF  JPA 

The  preceding  section  of  this  report  introduced  the  concepts 
of  the  hybrid  job  performance  aid  (HJPA)  and  enriched  hybrid  job 
performance  aid  (EHJPA) . The  aid  form  used  to  illustrate  these 
discussions  was  actually  one  of  the  several  possibilities  available 
for  implementing  these  concepts.  The  following  discussions  define 
the  concepts  more  specifically  and  show  candidate  approaches  to 
hybridization  and  to  enrichment.  Subsequent  experimentation  will 
evaluate  these  candidate  approaches  to  determine  those  amenable 
to  additional  development  and  test. 

Development  of  HJPA  Candidates 


The  concept  of  the  hybrid  JPA  is  a promising  candidate  for 
inclusion  in  the  Integrated  Personnel  System  (IPS)  because  it: 

1.  Permits  inexperienced  technicians  to  perform  complex 
troubleshooting  without  continuous  supervision. 

2.  Allows  postponement  or  redistribution  of  the  theory-based 
training  which,  historically,  has  been  given  at  the  beginning  of 
a new  enlistee's  career. 

3.  Has  the  potential  to  increase  aid  utilization  in  all 
stages  of  a technician's  career. 

Before  these  objectives  can  be  pursued,  the  hybrid  must  be 
defined  more  precisely.  Specifically,  it  is  necessary  to 
identify  the  population  of  directive  and  deductive  formats. 

Using  these  formats,  hybrids  can  be  developed  to  verify  that  the 
formats  are  feasible,  and  to  identify  specific  HJPA  aspects  for 
evaluation.  These  topics  are  discussed  below. 

Available  Formats 

Table  1 identified  26  presentation  techniques  that  are, 
theoretically,  available  for  use  in  developing  hybrids.  Results 
of  more  recent  studies  (Shriver  & Hart,  1975;  Booher,  1978)  have 
expanded  this  list.  At  present,  there  are  more  than  100  presen- 
tation techniques  that  could  be  considered  as  potential  hybrid 
aid  elements.  Clearly,  this  number  of  aid  elements  must  be 
reduced  to  a more  manageable  size  before  the  hybrid  can  be 
considered  a practical  part  of  the  IPS. 

The  work  of  Post,  Price,  and  Diffley  (1976)  provides  a 


basis  for  extracting  basic  aid  formats  from  the  more  than  100 
Drpspnhatinn  techniaues.  The  extraction  Drocess  relies  on  the 


type  of  behavior  required  of  the  JPA  user.  The  top  of  Figure  3 
lists  two  fundamentally  different  types  of  behavior  possible  for 
troubleshooting  tasks--instruction  following  and  decision  making. 
These  behaviors  are,  in  turn,  divided  into  two  subtypes,  resulting 
in  four  different  behaviors  required  of  JPA  users.  A basic  JPA 
format  is  matched  to  each  different  behavior,  as  shown  at  the 
bottom  of  Figure  3.  These  basic  JPA  formats  were  considered  in 
the  present  study  as  the  JPA  elements  that  could  be  used  or  com- 
bined to  form  a hybrid  JPA;  for  example,  by  combining  a fully 
proceduralized  JPA  with  a schematic.  Before  formal  consideration, 
however,  it  was  necessary  to  verify  that  each  basic  aid  format  had 
undergone  sufficient  field  or  laboratory  testing  to  qualify  as  an 
integral  part  of  the  Integrated  Personnel  System. 
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Figure  3.  Basic  aid  formats  and  user  behaviors. 

Verification  of  Basic  Aid  Formats 

The  aid  forms  considered  for  hybridizing  must  be  proven 
individually  before  they  can  be  used  in  combination.  This  means 
that  the  basic  aid  forms  to  be  considered  for  hybridization  must 
have  been  verified  through  at  least  laboratory  and  preferably 
field  test  evaluations.  Table  3 lists  a key  verification  project 
for  each  of  the  five  basic  aid  forms  discussed  in  Available 
Formats.  Thus,  all  five  formats  may  be  considered  for  hybridizing, 


Table  3 


Test  Data  Available  to  Verify  Basic  Aid  Formats 


Basic  Aid  Format 

Test  Data  Reference 

Fully  Proceduralized  JPAs 

Partially  Proceduralized  JPAs 

Maintenance  Dependency  Charts 
Functional  Flow  Logic  Diagrams 

Schematics 

Potter  & Thomas,  1976  (C-141) 

Horn,  1973  (AWG-10) 

Horn.  1973  (AWG  101 

Atchley  & Lehr,  1964,  Losee,  et  el.,  1962 
Atchley  & Lehr,  1964;  Losee,  et  al.,  1962 
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Identification  of  Specific  HJPA  Candidates 


Figure  4 represents  10  hybrids  that  can  be  formed  from 
the  basic  aid  formats^  being  considered.  Each  numbered  cell  of 
Figure  4 represents  a possible  (not  necessarily  desirable)  hybrid. 
For  example,  cell  6 represents  a hy’  'id  formed  by  combining  the 
partially  proceduralized  job  performance  aid  (PPJPA)  with  the 
functional  flow  logic  diagram  (FFLD) . 


The  aid  forms  listed  in  the  left-hand  column  represent 
the  element  the  user  would  rely  on  for  his  initial  performance 
attempts.  The  aid  forms  shown  in  the  header  of  Figure  4 are 
those  to  which  the  user  would  progress. 
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Figure  4.  Representation  of  hybrid  possibilities 
arising  out  of  five  basic  aid  formats. 


At  this  stage  of  the  hybrid  aid  development,  there  are 
ten  candidate  hybrids.  The  candidates  were  evaluated  to  deter- 
mine which  ones  to  field  test. 


The  MDC  and  the  FFLD  require  the  same  behavior  of  the  JPA 
user.  The  physical  features  of  these  aid  forms,  however,  are 
so  different  that  each  was  retained  as  a separate  aid  type. 
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Development  of  EHJPA  Candidates 


Specific  EHJPA  candidates  were  defined  by  two  dimensions: 

(1)  the  type  or  purpose  of  the  enrichment  message,  and  (2)  the 
means  or  format  used  to  make  the  enrichment  available  to  the 
user.  Each  of  these  dimensions  is  discussed  below,  followed 
by  an  identif ication  of  the  EHJPA  candidates  they  define. 

Definition  of  Enrichment  Types 

By  definition,  the  enrichment  must  relate  to  and  possibly 
go  beyond  the  content  of  the  two  elements  of  a hybrid  JPA.  On 
this  basis,  the  present  study  identified  three  types  of  enrichment 

1.  Transition--Any  information  designed  to  help  the 
hybrid  user  transition  from  the  directive  to  the  deductive  aid 
form. 

2.  System  Understanding--Information  about  the  operation 
of  the  hardware  represented  in  the  deductive  aid  of  the  hybrid. 

3.  Theory-Information  about  the  operation  of  or  prin- 
ciples underlying  the  class  of  equipment  to  which  the  present 
hardware  belongs. 

Delivery  Formats 

Research  by  the  Navy  (Smillie,  1978)  identified  three 
"formats"  by  which  enrichment  information  can  be  made  available 
to  the  user  of  a hybrid  aid. 

1.  Interpretive--A  format  that  provides  the  enrichment 
message  as  an  integral  part  of  the  hybrid  aid  (i.e.,  on  the  same 
sheet  of  paper) . 

2.  Basic  Reference  Cues — A technique  whereby  either  or 
both  of  the  hybrid  elements  may  contain  notes  that  refer  the 
user  to  another  source  document  (e.g.,  another  maintenance  manual 
or  a Rate  Training  Manual) . 

3.  Extension  Training  Reference  Cues--A  technique 
whereby  either  or  both  of  the  hybrid  elements  may  contain  notes 
which  refer  the  user  to  a personalized  training  source  available 
at  the  local  level  ("personalized"  excludes  local  classroom 
activities;  "local  level"  excludes  formal,  institutional  training) 


Identification  ol  Specific  EilJPA  Candidates 


Figure  5 combines  the  dimensions  of  enrichment  type  and 
delivery  format  into  a matrix  whose  cells  define  the  specific 
EHJPA  concepts  evaluated  in  this  report.  To  illustrate,  the 
enriched  hybrid  aid  which  appeared  as  Figure  2 in  the  enrichment 
concept  discussion  would  be  associated  with  cell  #1  because  the 
purpose  of  its  enrichment  message  is  to  help  the  user  to  tran- 
sition from  directive  to  deductive  modes  of  troubleshooting;  and 
the  enrichment  message  would  appear  as  an  integral  part  of  one  of 
the  hybrid  elements — in  this  case,  the  directive  element. 

These  nine  types  of  EHJPAs  were  evaluated  to  determine 
which  offered  the  most  promise  to  the  Integrated  Personnel  System 
The  results  of  this  evaluation  are  discussed  in  the  next  section. 
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Figure  5.  Matrix  to  define  EHJPA  candidates.* 


♦Combinations  of  these  nine  types  of  EHJPAs  are  also  possible 
For  example,  types  1 and  4 and  types  1,  4,  and  5 are  possible 
approaches  to  enrichment.  However,  the  evaluation  focused  on  the 
nine  basic  types. 
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EVALUATION  OF  THE  HYBRID  JPA  AND 
ENRICHED  HYBRID  JPA  CANDIDATES 


Evaluation  of  the  HJPA  Candidates 

The  ten  HJPA  candidates  were  pared  to  two  promising  candidates 
through  subject  assessment.  These  candidates  were  then  evaluated 
by  pilot  testing. 

In  the  earlier  discussion  of  the  hybrid  concept.  Figure  4 
used  five  basic  aid  formats  to  define  ten  hybrid  possibilities. 
This  figure  is  repeated  here  for  convenience  and  appears  as 
Figure  6.  The  ten  hybrid  possibilities  were  rated  by  project 
personnel  for  the  purpose  of  eliminating  those  candidates  that 
did  not  warrant  further  investigation.  The  rating  criteria  c*nd 
outcomes  are  discussed  below. 


Basic  Aid  Formats 


B»ic  Aid  Formats 

FPJPA 

< 

CL 

“1 

& 

MDC 

FFLO 

FPJPA 

i* 

2 

3 

4 

PPJPA 

' 

b 

6 

7 

MDC 

y 

B 

9 

FFLO 

> 

: 

■ 

! 

10 

Schematic 

Iflfl 

• V 

-i 

\ 

• 

*C«ll  entries  refer  to  transition  from  row  heading  to  column  heading 


Figure  6.  Representation  of  hybrid  possibilities 
arising  out  of  five  basic  aid  formats. 

Rating  Criteria 

Two  criteria  were  considered  in  assessing  the  ten  hybrid 
aid  candidates.  A major  objective  of  the  hybrid  is  to  foster 
early  productivity  in  inexperienced,  minimally  trained  technicians. 
Thus,  one  criterion  used  for  assessing  hybrid  feasibility  was  the 
performance- fostering  power  of  the  directive  element.  Since 
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another  objective  of  the  hybrid  concept  is  to  focus  on  transi- 
tioning from  one  aid  to  another,  the  second  criterion  was  the 
direction  and  ease  of  the  transition.  Table  4 lists  these  two 
criteria,  along  with  the  possible  outcomes  associated  with  each 


HJPA  Rating  Results 

The  six  rating  outcomes  shown  in  Table  4 are  assigned 
as  appropriate  to  each  of  the  HJPA  candidates.  These  assignments 
are  shown  in  Figure  7,  along  with  a code  that  indicates  whether 
the  hybrid  possibility  is  rejected  (R)  or  considered  feasible 
(F) . Thus,  R-1,4  indicates  that  the  hybrid  possibility  is 
rejected  because  (1)  the  simpler  element  is  stronger  or  more 
detailed  than  necessary  for  the  target  population  and  (4)  the 
transition  is  too  simple  and  of  the  wrong  type. 

The  assumptions  made  in  arriving  at  the  outcomes  shown 
in  Figure  7 are  listed  below: 

1.  The  HJPA  is  primarily  an  intermediate  aid  form, 
expected  to  be  used  by  technicians  newly  qualified  at  this  level. 

2.  During  their  novice  work,  technicians  obtained 
substantial  hardware  geography  familiarization  (e.g.,  test  point 
and  component  location)  even  though  their  assignments  were  con- 
fined to  preventive  and  nontroubleshooting  maintenance  tasks. 

3.  Prior  to  assignment,  technicians  are  given  task- 
oriented  training  on  test  point  location,  test  equipment  and 
prime  system  setup,  and  test  equipment  and  aid  usage. 


Table  4 

HJPA  Rating  Criteria 


Criteria 


A Performance-fostering  powei  of  the 
directive  element 


B Direction  and  ease  of  transition 


Outcome  Possibilities 


1 Stronger  than  necessary  for  the  target  population 

2 Appropriate  power  for  the  target  population 

3 Insufficient  power  for  the  target  population 

4 Wrong  type,  merely  rote  learning,  or  learning 
within  an  aid  type 

5 Correct  type,  between  aid  forms,  but  not  too 
large  a transition 

6 Too  demanding;  spans  are  too  large  or  too 
complex 


. . 4 

4.  Technicians  who  use  HJPAs  are  Category  3a  and  above. 

Changes  in  this  assumption  would  cause  major  shifts  in  the  eval- 
uation outcomes. 
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Figure  7.  Results  of  applying  rating  criteria 
to  10  hybrid  possibilities. 

Observations  about  the  rating  results  shown  in  Figure  7 
are  as  follows: 

1.  The  PPJPA  -*■  FFLD  and  the  PPJPA  -*■  MDC  appear  to  be 
the  best  candidates  against  the  criteria  used. 

2.  The  code  1 assignments  made  to  FPJPA  possibilities 
are  judgmental  and  based  on  all  assumptions  about  the  target 
population.  Any  changes  in  the  assumptions  could  change  the 
code  1 assignments. 

3.  The  two  code  4 assignments  reflect  an  undesirable 
transition  within  a behavior  type,  as  opposed  to  between  behavior 
types . 


4 

Category  3a  personnel  are  defined  as  those  personnel  having 
an  average  percentile  score  between  64  and  49  on  the  word  know- 
ledge, arithmetic  reasoning,  and  spatial  perception  subtests  of 
the  Armed  Services  Vocational  Aptitude  Battery. 
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4.  The  code  6 assignments  are  less  a rejection  of  the 
higher  order  deductive  aid  (schematics,  system  descriptions)  and 
more  a reflection  of  the  incompleteness  of  the  hybrid  concept. 

For  example,  the  two  stage  PPJPA  -*■  FFLD  might  well  be  expanded 
to  a third  stage  to  yield  a PPJPA  -*•  FFLD  -*■  Schematics,  thereby 
breaking  the  transition  into  smaller,  more  achievable  segments. 

5.  The  two  possibilities  that  start  with  Maintenance 
Dependency  Charts  (MDCs)  were  assigned  a code  2 on  the  basis 
of  an  experiment  (Horn,  1973)  that  shows  MDCs  to  be  inferior 
to  PPJPAs  in  terms  of  performance-fostering  power. 

6.  An  additional  consideration  of  the  code  4 assignments 
to  the  MDC  -*•  FFLD  possibility  is  based  on  the  fact  that  both  aid 
forms  require  decisions  without  knowledge  of  correct  response 
being  available  to  users. 

On  the  basis  of  the  foregoing  assessment,  two  HJPA 
candidates  were  chosen  for  further  evaluation  (discussed  below) . 
Both  candidates  use  a partially  procedural ized  aid  as  the  simpler, 
directive  element.  One  HJPA  candidate  transitions  to  an  FFLD 
element;  and  the  other,  to  an  MDC  element. 

Pilot  Test  of  HJPAs 

A pilot  test  was  conducted  on  the  survivors  of  the  HJPA 
rating  to  determine  the  technical  feasibility  of  transitioning 
from  one  to  the  other  aid  form.  This  experiment  and  its  results 
are  described  below. 

Test  Design.  The  test  design  is  presented  in  graphic 
form  in  Figure  6.  A description  of  the  treatments  and  a rationale 
for  their  arrangement  is  provided  in  the  following  section.  The 
test  design  is  set  up  to  permit  a comparison  of  improvement  over 
trials  for  the  three  treatments.  While  the  test  is  designed  to 
show  feasibility  of  the  hybrid  concept,  it  was  felt  that  some 
useful  comparisons  would  emerge  with  the  addition  of  the  enriched 
version  of  the  hybrid. 
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Figure  8.  Design  of  HJPA  experiment. 
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All  subjects  were  briefed  on  the  test  materials  and  task 
operations.  A test  monitor  was  present  during  all  trials  so  that 
misunderstandings  could  be  resolved,  if  necessary. 

The  pretest  was  conducted  using  only  the  deductive  element 
of  the  hybrid  aid  (Functional  Flow  Logic  Diagram  or  Maintenance 
Dependency  Chart) . The  learning  trials  were  conducted  using  the 
various  hybrid  arrangements  while  the  post-test  employed,  again, 
only  the  deductive  aid.  (The  paradigm  is  that  of  standard 
transfer-of- training  test  procedures.)  Subjects  were  given  a 
10-minute  time  limitation  per  task  (trial) . 

Subjects . Two  groups  of  subjects  were  used.  Twenty-eight 
were  undergraduate  students  at  Northern  Virginia  (NOVA)  Community 
College  and  seven  were  employees  of  BioTechnology , Inc.  (BTI ) . 

The  28  students  were  randomly  assigned  to  two  equal-sized  groups 
of  14  each.  These  two  groups  experienced  the  PPJPA/FFLD  and  the 
enriched  PPJPA/FFLD  conditions.  For  logistical  reasons,  all  sub- 
jects in  the  unenriched  PPJPA/MDC  condition  were  BTI  employees. 

Task  Characteristics.  The  tasks  involved  paper-and-pencil 
versions  of  electronic  troubleshooting  tasks.  They  were  derived 
from  the  actual  troubleshooting  operations  for  the  AWG-10  missile 
system.  Each  task  was  performed  using  the  same  hybrid  aid.  The 
variation  between  tasks  was  in  the  "correct  answer";  that  is,  the 
component  with  the  malfunction  was  different  for  each  task.  Each 
subject  experienced  the  same  task  sequence,  that  is,  the  order  of 
correct  answer  was  the  same  throughout  all  conditions. 

Description  of  Test  Materials.  PPJPA  -*•  FFLD.  An  EHJPA 
was  added  to  the  experiment  to  show  the  contribution  made  by  the 
addition  of  an  enrichment  message.  Figure  9a  illustrates  the 
enriched  version  of  the  partially  proceduralized  aid.  Subjects 
assigned  to  the  straight  hybrid  group  performed  with  the  aids 
shown  in  Figure  9a  but  without  the  information  shown  in  the 
brackets,  and  subjects  assigned  to  the  enriched  group,  with  the 
PPJPA  shown  in  Figure  9a.  Figure  9b  shows  the  deductive  aid  used 
by  subjects  in  both  groups. 

PPJPA  ► MDC . Figures  10a  and  10b  represent  the  PPJPA 
combined  with  the  MDC  as  the  deductive  element  of  the  HJPA. 

(Only  the  lower  part  of  the  MDC  relates  to  the  PPJPA  shown.) 

Pilot  Test  Results.  The  experiment  used  two  performance 
measures:  percent  correct  responses  (accuracy)  and  delay, 

measured  as  seconds  per  test  point.  The  findings  are  summarized 
in  Figures  11  and  12.  Regarding  accuracy,  the  pretest  results 
show  similar  percent  correct  scores  for  the  three  conditions. 

The  enriched  hybrid  using  a functional  flow  diagram  gains  most 
in  the  learning  sequence,  while  the  straight  hybrid  using  an  MDC 
does  not  show  any  accuracy  improvements. 
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Figure  9a.  Enriched  hybrid  JPA  candidate  (directive  element) . 


Figure  9b.  Enriched  hybrid  JPA  candidate  (deductive  element) . 


Figure  10a.  PPJPA  -*•  MDC  hybrid  candidate  (directive  element). 


Figure  10b.  PPJPA  -*■  MDC  hybrid  candidate  (deductive  element) 
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Figure  11.  Percent  of  correct  responses 

at  three  stages  of  the  test  sequence. 
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Figure  12.  Delay  at  three  stages  of  the  test  sequence 


With  regard  to  delay,  all  aid  forms  start  with  near 
equivalency.  During  the  learning  trials,  the  enriched  hybrid 
with  functional  flow  shows  improvement  (as  does  the  unenriched 
hybrid)  while  the  straight  hybrid  using  an  MDC  shows  no  improve- 
ment. 


Implications . Several  cautions  are  in  order.  First,  the 
experiment  was  a pilot  test,  intending  only  to  show  the  relative 
promise  or  feasibility  of  the  three  aid  forms.  Accordingly,  only 
descriptive  statistics  are  presented.  Second,  the  comparability 
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of  subjects  can  be  questioned.  In  this  regard,  however,  it 
should  be  noted  that  the  use  of  BioTechnology  personnel,  because 
of  their  relatively  higher  level  of  education,  should  have  given 
the  unenriched  hybrid  using  MDC  an  advantage  over  the  other  two 
conditions . 

The  findings  are  only  suggestive  implications,  but  even 
at  this  level  of  inference,  the  conclusions  are  provocative. 
Specifically : 

1.  Hybridization  alone  has  a beneficial  effect  on 
performance. 


2.  The  hybrid  transitioning  to  an  MDC  is  the  least 
productive  of  the  aids  tested. 

3.  The  enrichment  concept,  at  least  at  this  transition 
level,  seems  to  offer  promise. 

Evaluation  of  EHJPA  Candidates 


Two  subjective  evaluations  were  made  of  the  nine  EHJPA 
candidates.  In  the  first,  six  JPA  research  experts  were  asked 
to  rate  the  nine  candidates,  knowing  that  those  receiving  low 
ratings  were  to  be  eliminated.  In  the  second,  three  former 
military  technicians  were  asked  to  perform  a similar  evaluation. 
The  procedures,  criteria,  and  results  of  these  evaluations  are 
described  below. 

Procedures 


Raters . As  indicated  above,  the  evaluation  employed  two 
sets  of  raters.  The  individuals  in  the  set  of  six  raters  were 
chosen  because  they  are  putative  experts  in  the  field  of  job 
performance  aiding.  Those  in  the  set  of  three  raters  were 
chosen  because  of  their  experience  in  using  and  supervising  the 
use  of  job  aids  in  the  field. 

Criteria  for  Rating.  Three  criteria  were  considered  in 
rating  the  nine  EHJPA  candidates.  The  most  important  was  volition, 
which  refers  to  (1)  the  likelihood  that  a user  would  voluntarily 
avail  himself  of  the  enrichment  information,  and  (2)  given  that 
the  user  does  avail  himself  of  the  enrichment  message,  whether 
he  would  consider  the  effort  worthwhile. 

The  second  criterion  was  labeled  expected  work  output. 

The  issue  in  question  was  whether  the  particular  enrichment 
strategy  might  tend  to  impair  timely  and  accurate  completion 
of  the  immediate  task.  Over  an  entire  action  sequence,  the 
effect  of  enrichment  could  be  to  produce  a noticeable  delay  in 
task  completion.  Even  worse,  the  enrichment  information  could 
be  a distraction  to  the  user.  If  that  were  the  case,  more  errors 
might  be  committed. 
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The  third  criterion  was  the  expected  level  of  learning 
for  the  information  provided  via  a given  enrichment  strategy. 

Rating  Scales.  For  each  of  the  criteria,  a rating  scale 
was  developed  that  consisted  of  two  subscales.  For  volition, 
the  subscales  were  probability  of  exposure  and  reward  value. 
Probability  of  exposure  means  the  likelihood  that  the  technician 
will  read  or  study  the  message.  Reward  value  refers  to  a 
conceptual  cluster  of  possible  reinforcements  that  range  from 
the  satisfaction  of  understanding  the  purpose  behind  the  task 
being  performed  to  the  sense  of  personal  effectiveness  that  is 
associated  with  doing  the  task  well.  Among  the  cluster  of 
possible  reinforcements,  however,  the  most  crucial  one  for  rating 
purposes  should  be  whether — or  the  probability  that — the  enrich- 
ment is  judged  to  contribute  to  the  feeling  of  independence  or 
autonomy  on  the  part  of  the  technician  in  the  task  situation. 

For  expected  work  output,  the  subscales  were  time 
(i.e.,  expected  delay)  and  error  rate.  Reading  the  enrichment 
message  takes  some  time  that  might  otherwise  be  used  in  direct 
work  on  the  task.  Consequently,  some  delay  is  built  in.  The 
issue  is  how  much  delay,  as  well  as  whether  any  recapture  of  time 
is  likely  due  to  the  technician’s  enhanced  confidence  or  other 
positive  contribution  from  the  enrichment  messages.  Concern  for 
error  rate  is  based  on  the  possibility  that  the  enrichment 
message  could  act  as  a distractor  in  breaking  the  technician's 
concentration  on  the  task  at  hand.  One  assumption  is  that,  if 
the  technician's  attention  is  diverted,  some  increase  in  proba- 
bility of  error  will  occur. 

The  two  subscales  for  expected  level  of  learning  were  the 
amount  or  quality  of  task/career  relevant  material  learned  and 
retention  over  time.  If  the  enrichment  information  is  meaningful 
to  the  technician's  career  or  to  the  task  at  hand,  then  the 
degree  to  which  the  information  is  used  would  be  a valid  means  of 
rating  EHJPAs . Assessing  the  technician's  retention  of  the 
information  would  be  another  method  of  determining  the  usefulness 
of  an  EHJPA. 

Expert  Rating.  The  six  experts  in  the  first  set  of 
raters  were  briefed  on  the  nature  of  the  enrichment  strategies 
and  the  rating  criteria.  Each  expert  was  given  a briefing  book 
that  contained  either  a concrete  example  or  a written  description 
of  each  candidate  enrichment  strategy. 

After  hearing  the  briefing,  discussing  ambiguities,  and 
examining  the  examples,  the  raters  were  asked  to  independently 
complete  rating  forms  that  corresponded  to  each  of  the  subscales. 
These  forms  were  an  integral  part  of  the  briefing  book.  An 
example  of  such  a form  is  presented  in  Figure  13. 


32 


Form  A-l 

Probability  of  Exposure  to  Enrichment  Message 


Enrichment  Strategy 

Indicate  on  the  scales  below  your  estimate  of  the  prob- 
ability that  the  typical  technician  will  read/study  the  en- 
richment message 

1 1 1 1 1 

Enrichment  Type 

Message  Format 

Interpretive 

.25 

.50 

.75 

1.00 

T ransition 

Basic  Reference  Cues 

1 

1 

1 

0 

.25 

.50 

.75 

Too 

Extension  T raining 

1 

1 

1- 

1 

1 

Reference  Cues 

0 

2 5 

.50 

.75 

1.00 

1 1 1 1 1 

1 n terpret  ive 

0 

.25 

.50 

.76 

1 00 

System 

Basic  Reference  Cues 

1 

i 

1 

1 

Understanding 

0 

.25 

.60 

.76 

1.00 

Extension  Training 

1 

1 

1 

1 

Reference  Cues 

0 

.BO 

.76 

Too 

Interpretive 

1 1 1 1 1 

JB 

iBO 

1 DO 

Theory 

Basic  Reference  Cues 

k — 

J6 

± 

A 

T56 

Extension  Training 

1 

1 

i 

1 

J 

Reference  Cues 

I 

1 

I 

0 

.26 

50 

.76 

1.00 

Figure  13.  Sample  of  rating  forms. 
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One  week  after  the  six  experts  made  their  initial  rating, 
a second  meeting  was  convened  to  review  the  results  and  discuss 
the  differences  between  raters.  As  might  be  expected,  the 
discussion  revealed  some  variation  between  judges  in  their  inter- 
pretation of  certain  criteria  and  their  assumptions  about  how  the 
EHJPAs  might  be  used. 

The  discussion  was  structured  by  taking  each  scale  in 
turn  and  identifying  anomalies--if  any--in  the  ratings  assigned 
to  each  candidate.  The  rationale  for  any  anomalous  rating  was 
elicited  and  discussed  in  such  a way  that  all  raters  understood 
the  reasoning  behind  the  anomalous  rating (s).  Following  this 
discussion,  the  experts  were  asked  to  rerate  the  candidate  EHJPA 
options . 


User  Rating.  As  suggested  above,  the  procedure  was  much 
the  same  for  the  users  as  for  the  experts.  Due  to  a limitation 
of  time,  however,  only  one  rating  session  was  held  and  only  four 
of  the  six  subscales  were  used.  These  four  subscales--volition , 
time,  error,  and  learning--were  considered  as  providing  the  most 
relevant  information. 

The  three  users  were  briefed  as  a group  and  given  the 
briefing  book  containing  examples/descriptions  of  the  job  aid 
materials  for  each  enrichment  strategy.  Ambiguities  regarding 
concept,  criteria,  or  scales  were  resolved  in  open  discussion. 

Results 


Tables  5 and  6 present  the  average  ratings  given  to  each 
of  the  nine  EHJPA  candidates  by  the  experts  and  the  users.  These 
data  were  used  in  a candidate  evaluation  designed  to  select 
enrichment  candidates  for  additional  testing.  The  evaluation  was 
conducted  in  two  steps  described  below. 

Step  1:  Top  Three  Candidates.  The  user  and  expert 
ratings  for  a given  strategy  and  a given  scale  were  examined  to 
see  whether  that  strategy  was  among  the  top  three.  For  example, 
Figure  14,  which  provides  the  A-l  rating  of  the  users  for  all 
nine  enrichment  strategies,  shows  that  candidates  1,  5,  and  7 
received  the  three  highest  ratings.  Performing  this  analysis  on 
all  the  subscales  in  the  user/expert  ratings  yields  the  results 
shown  in  Figure  15.  Candidates  1,  5,  7,  and  9 accumulated  the 
most  "hits'*  (four  or  greater)  in  this  assessment;  accordingly, 
they  are  considered  for  additional  testing  at  this  stage  of  the 
evaluation . 
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Table  5 

Experts'  Ratings  on  Each  Scale 


Interpretive 


Basic  Reference 
Cues 


Extension  Training 
Reference  Cues 


TRANSITION 


© 


© 


© 


A-1 

86 

A-1 

A 2 

6 9 

A-2 

B-1 

5.3 

B-1 

B 2 

6.2 

B 2 

Cl 

7.1 

C-1 

C 2 

5.9 

C-2 

© 


5 4 

A-1 

3.5 

5.0 

A-2 

4.3 

4.0 

B-1 

6.7 

4.1 

B-2 

4.1 

5.2 

C-1 

5.3 

5.2 

C-2 

6.1 

© 


SYSTEM 

UNDERSTANDING 


A-1 

7.2 

A-1 

A-2 

6.3 

A-2 

B-1 

4.3 

B-1 

B-2 

4.4 

B-2 

C-1 

4.5 

C-1 

C-2 

5.4 

C-2 

© 


5.8 
6.2 

3.8 
5.4 
55 
4.6 


A-1 

A-2 

B-1 

B-2 

C-1 

C-2 


© 


2.9 

3.6 

6.2 

5.1 
5.5 

7.1 


© 


THEORY 


A-1 

6.2 

A-2 

4.2 

B-1 

4.1 

B-2 

4.6 

C-1 

2.8 

C-2 

3.1 

A-1 

38 

A-2 

3.6 

B-1 

4.5 

B-2 

4.7 

C-1 

3.1 

C-2 

2.3 

A-1 

2.6 

A-2 

5.0 

B-1 

6.2 

B-2 

6.2 

C-1 

6.0 

C-2 

5.3 

SCALES 

A1  ■ Volition 
A2  - Reward 


Step  2:  Lowest  Three  Candidates.  Figure  16  shows  the 
results  of  applying  the  procedure  of  step  1 to  determine  whether 
a candidate  was  rated  as  one  of  three  least  desirable  enrichment 
candidates.  For  example,  for  the  subscale  in  Figure  14, 
candidates  2,  3,  and  6 received  the  lowest  ratings.  As  shown 
(Figure  16),  candidates  2,  3,  6,  and  7 accumulated  the  most 
"hits,"  which  is  indicative  of  a least-desirable  status.  The 
complement  of  these  least  desirable  candidates  comprises  candi- 
dates 1,  4,  5,  8,  and  9.  Comparing  these  results  with  those  of 
the  procedure  in  step  1 yields  only  three  candidates  that  are 
members  of  both  sets;  viz.,  candidates  1,  5,  and  9. 


On  the  basis  of  this  analysis  of  expert  and  user  ratings, 
the  following  EHJPA  candidates  are  recommended  for  additional 
testing : 

1.  Candidate  1:  Transition--Interpretive  (solid  support 
from  the  expert  raters) . 

2.  Candidate  5;  System  Understanding — Basic  Reference 
Cue  (modest  support  from  the  expert  raters) . 

3.  Candidate  9:  Theory — Extension  Training  Reference 
Cue  (modest  support  from  both  the  expert  and  user  raters) . 
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CONCLUSIONS 


Hybrid  JPA 

Combining  directive  and  deductive  aids  into  a hybrid 
troubleshooting  JPA  is  intended  to  teach  naive  technicians  how 
to  troubleshoot  relying  only  on  the  deductive  aid.  The  evalu- 
ation studies  conducted  during  this  project  indicate  that  this 
goal  is  achievable.  The  evidence  to  support  this  contention  is 
in  the  form  of  results  of  paper-and-pencil  malfunction  problems 
that  subjects  were  asked  to  solve.  Subjects  achieved  a low 
accuracy  in  their  attempt  to  solve  the  initial  problem  supported 
by  only  a deductive  aid.  After  several  learning  trials,  performing 
with  the  hybrid  JPA,  subjects  again  used  only  the  deductive  aid 
to  solve  a "test  problem,"  demonstrating  considerable  improvement 
in  their  accuracy  scores. 

Enriched  Hybrid  JPA 

Enrichment,  when  used  to  enhance  the  effect  of  hybridization 
(referred  to  as  Transition  enrichment) , appears  feasible.  The 
evidence  to  support  this  contention  is  in  the  form  of  subject 
performance  on  paper-and-pencil  troubleshooting  problems. 

Subjects  performing  with  enriched  hybrid  aids  learned  to  trouble- 
shoot with  deductive  aids  twice  as  effectively  as  subjects 
performing  with  identical  but  unenriched  hybrids. 

In  addition,  enrichment  of  hybrids  appears  feasible  when  it 
is  used  to  convey  career-relevant  knowledge  about  the  hardware  at 
hand  (called  System  Understanding)  as  well  as  knowledge  of  the 
principles  governing  classes  of  hardware  (called  Theory) . Evidence 
to  support  this  contention  is  in  the  form  of  expert  and  user 
opinion  ratings  of  various  enrichment  candidates.  Ratings  were 
solicited  on  (1)  likelihood  of  aid  use,  (2)  adverse  effect  on 
task  performance,  and  (3)  learning  effects. 
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RECOMMENDATIONS 


The  evidence  to  support  the  findings  presented  above  was 
in  the  form  of  a pilot  test  (e.g.,  small  sample  size,  part  task 
representation,  nonrepresentative  subjects) , and  expert  opinions 
of  JPA  research  experts  and  JPA  user  experts.  Prior  to  an 
extensive  field  test  of  the  enriched  hybrid  JPA,  a more  rigorous 
demonstration  of  its  feasibility,  under  more  representative  con- 
ditions, is  required. 

The  hybrid  and  enrichmment  aid  concepts  were  delimited  in 
the  research  reported  here,  and  are  expected  to  undergo  rigorous 
experimentation  to  prove  their  technical  feasibility.  In  addition, 
analytical  work  is  required  to  define  key  operational  considera- 
tions. These  are: 

1.  Aid  Applicability.  Several  aspects  of  the  aid  forms  must 
be  defined  before  they  can  be  considered  ready  for  even  limited 
demonstration  projects.  For  example,  the  Theory-Extension 
Training  Reference  Cues  strategy  (heavily  favored  by  the  user 
poll  taken  during  this  project)  requires  considerable  work  on  how 
to  gauge  the  user's  qualifications  in  order  to  control  his  exposure 
to  enrichment  messages. 

2.  Head  Book  Trade-Off.  The  interface  of  all  enriched  JPAs 
with  formal  and  extension  training  requires  definition.  This 
interface  should  cover  the  degree  to  which  mastery  of  a particular 
EH JPA  qualifies  a user  for  formal  schooling  as  well  as  the  type  of 
schooling  required  to  qualify  a user  for  a particular  EHJPA. 

3.  IPS  Interface.  An  objective  of  the  IPS  is  to  facilitate 
advancement  along  any  of  several  career  path  options.  The  extent 
to  which  EH JPAs , individually  and  in  combination  with  formal 
training,  qualify  a user  for  career  milestones  needs  to  be 
established.  For  example,  mastery  of  some  of  the  EHJPAs  appears 
to  be  related  to  the  requirements  of  the  Personnel  Qualification 
System;  and  mastery  of  the  theory  types  of  enrichment  appears  to 
relate  to  a user's  preparedness  to  take  the  advancement  in  Rate 
exams.  The  full  potential  of  JPAs  in  the  IPS  will  not  be  under- 
stood until  relationships  such  as  these  are  identified  and  defined. 
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